(19) 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



(12) 



(11) EP1 016 991 A2 

EUROPEAN PATENT APPLICATION 



(43) Date of publication: 

05.07.2000 Bulletin 2000/27 

(21) Application nunnber: 99310597.2 

(22) Date of filing: 24.12.1999 



(51) Intel 7; G06F 17/30 



(84) Designated Contracting States: 

AT BE CH CY DE DK ES Fl FR GB GR IE IT LI LU 
MC NL PT SE 

Designated Extension States: 
AL LT LV MK RO SI 

(30) Priority: 28.12.1998 JP 37274698 

(71) Applicant KABUSHIKI KAISHA TOSHIBA 
Kawasaki-shi, Kanagawa-ken 210-8572 (JP) 

(72) Inventors: 

• Hori, Osamu, c/o Intellectual Property Div. 
Minato-ku, Tokyo 105-8001 (JP) 



• Dol, Miwako, c/o Intellectual Property Div. 
Minato-ku, Tokyo 105-8001 (JP) 

• Sumita, Kazuo, c/o Intellectual Property Div. 
Minato-ku, Tokyo 105-8001 (JP) 

• Hirakawa, HIdeki, c/o Intellectual Property Div. 
Minato-ku, Tokyo 105-8001 (JP) 

(74) Representative: Midgley, Jonathan Lee 
Marks & Clerk 
57-60 Lincoln's Inn Fields 
GB-London WC2A 3LS (GB) 



(54) information providing method and apparatus, and information reception apparatus 



(57) By moving image analysis, acoustic/speech 
analysis, or text analysis for multimedia information in a 
database, feature data representing the type of informa- 
tion is acquired, and the feature data is stored into the 
database (103) added to the multimedia information. A 
search engine (105) extracts partial images of user's In- 
terest from the multimedia information on the basis of 



the feature data and a user profile data. A link section 
(106) associates the representative images (still imag- 
es) of the partial images with multimedia images and 
displays the list of representative images and feature 
data. Thus, only a portion of the user's concern is ex- 
tracted from an enormous amount of multimedia infor- 
mation, and individual information is selectively provid- 
ed in units of users. 
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Description 

[0001] The present invention relates to a multimedia 
information providing method and apparatus for provid- 
ing video, music, and text data to many and unspecified 5 
users through the Internet, etc., and a multimedia infor- 
mation reception apparatus for receiving the video, mu- 
sic, and text data. More particularly, the present inven- 
tion relates to a multimedia information providing meth- 
od and apparatus and a multimedia information recep- 
tion apparatus for selecting only information of user's 
interest from a number of multimedia information and 
providing individual information to the user. 
[0002] This application is based on Japanese Patent 
Application No. 10-372746, filed December 28, 1998, 
the entire content of which is incorporated herein by ref- 
erence. 

[0003] In recent years, growth of information infra- 
structures is boosting opportunities for distributing home 
many multimedia information through CATV (cable tel- 
evision broadcasting), digital satellite broadcasting, or 
the Internet. A variety of programs are provided, and the 
number of service channels has reached an order of 
several hundred or several thousand. Therefore, it is be- 
coming difficult for a user to appropriately select infor- 
mation from the several hundred or several thousand 
channels or tens of thousands or more programs in the 
channels. 

[0004] To solve this problem, a receiver device for au- 
tomatically recording programs of user's interest using 
the information of an electronic program list sent from a 
broadcasting station has been proposed (e.g., "video 
device " disclosed in Jpn. Pat. Appln. KOKAI Publication 
No. 7-135621). 

[0005] This proposed device selects programs that 
may be of interest tor a user from the Information of an 
electronic program list on the basis of keywords regis- 
tered In advance and automatically filters programs in 
units of users. 

[0006] To prepare an attractive program providing 
program, a program provider wants to know the types 
of programs viewed by viewers. Conventionally, a pro- 
vider raises monitors, lets them fill out a questionnaire, 
and gathers the results to know programs watched by 
the monitors. However, with the questionnaire of fill-out 
type, only rough information representing whether or not 
a viewer has watched a certain program can be ob- 
tained. 

[0007] In a conventional system for automatically se- 
lecting a program from an enormous number of pro- 
grams provided by a program provider in accordance 
with a personal taste, selection is just roughly done in 
units of programs. Consider a program such as a news 
show or a variety show. In such programs, one program 
is constructed by units of "topics" or "corners". Quite of- 
ten, user's interest is only in some of images in one pro- 
gram. However, in automatic recording in units ot pro- 
grams, one program is entirely selected and recorded 
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from the beginning to the last. The user cannot know the 
position of information of his/her actual interest unless 
he/she watches the entire program. Hence, even when 
a program is selected and recorded by filtering, the user 
must watch the recorded program from the beginning to 
the last, wasting the recording medium and user's time. 
[0008] Filtering may omit CMs contained in a pro- 
gram. When a broadcasted program is not a pay TV pro- 
gram but a free program for which the ad rate is the 
source of revenue, whether the viewers actually watch 
CMs or not is an important factor for the program pro- 
vider in soliciting advertisement. Hence, to exclude CMs 
from the program content poses a serious problem. 
[0009] In addition, conventional audience rating sur- 
vey is done in units of programs and is therefore insuf- 
ficient to precisely grasp the users' tastes and the like. 
[001 0] Accordingly, it is an object of the present inven- 
tion to provide the following information providing meth- 
od and apparatus, information reception apparatus, and 
data structure. 

[0011] It is the first object of the present invention to 
provide an information providing method and apparatus 
and an information reception apparatus capable of ap- 
prophately selecting and providing portions of user's ac- 
tual interest from a number of multimedia information 
instead of filtering in units of programs. 
[0012] It is the second object of the present invention 
to provide an Information providing method and appa- 
ratus and an Information reception apparatus capable 
of appropriately selecting and providing portions of us- 
er's actual interest from a number of multimedia infor- 
mation instead of filtering in units of programs, In which 
a commercial message that the program provider wants 
a viewer to watch is surely provided. 
[0013] It is the third object of the present invention to 
provide an information providing method and apparatus 
and an information reception apparatus capable of ap- 
propriately selecting and providing portions of user's ac- 
tual interest from a number of multimedia information 
instead of filtering in units of programs, in which user's 
viewing history is recorded, and a user profile represent- 
ing user's taste can be updated in accordance with the 
viewing history. 

[0014] It is the fourth object of the present invention 
to provide a data structure of describing the above user 
profile used in the information providing apparatus and 
the information reception apparatus. 
[0015] According to the present invention, there is 
provided an information providing method comprising: 

adding program feature data to multimedia informa- 
tion in units of parts of the multimedia information 
to form a program database; 
searching for partial information which accords with 
user profile data from the multimedia information 
based on matching between the user profile data 
and the program feature data; and 
providing the searched partial information to a user. 
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[0016] According to the present invention, there is 
provided an infornnation providing apparatus compris- 
ing: 

a first database configured to store multimedia in- 5 
formation; 

an analyze section configured to analyze the multi- 
media information stored in the first database using 
at least one analysis method of moving image anal- 
ysis, acoustic/speech analysis, and text analysis; io 
a second database configured to store program fea- 
ture data which is obtained in units of parts of the 
multimedia information or externally inputted; and 
a search engine configured to search for program 
feature data from the second database in accord- 
ance with user profile data, and select partial infor- 
mation from the multimedia information stored in 
the first database in accordance with searched pro- 
gram feature data. 

20 

[0017] According to the present invention, there is 
provided an information reception apparatus connected 
to an information providing server having a database 
which stores multimedia information and program fea- 
ture data which is an analysis result of at least one of 2S 
moving image analysis, acoustic/speech analysis, and 
text analysis or externally inputted, comprising: 

a search engine configured to search for prede- 
termined program feature data from the database and 
select partial information from the multimedia informa- 30 
tion stored In the database in accordance with searched 
program feature data. 

[0018] According to the present invention, there is 
provided an information describing method comprising: 

35 

classifying information items into plural groups of in- 
formation items relating to personal information of 
a user, some of the groups of information items In- 
cluding plural subgroups; and 

describing each Information items in the group or 40 

the subgroup in an order according to a priority of 
the information item which is determined for each 
user. 

[0019] According to the present invention, corre- -^5 
sponding partial information can be selected on the ba- 
sis of a user profile data. 

[0020] It is possible to select such a commercial mes- 
sage in accordance with the user profile data a commer- 
cial message that a program provider wants a viewer to 50 
watch if the commercial message is also stored in the 
database as in the same manner as the multimedia in- 
formation. 

[0021] This summary of the Invention does not nec- 
essarily describe all necessary features so that the in- 55 
vention may also be a sub-combination of these de- 
scribed features. 

[0022] The invention can be more fully understood 



from the following detailed description when taken in 
conjunction with the accompanying drawings, in which: 

FIG. 1 is a block diagram showing the basic ar- 
rangement of an information providing apparatus 
according to the first embodiment of the present in- 
vention; 

FIGS. 2A to 2E are views showing the data structure 
of a user profile; 

FIG. 3 is a flow chart showing the operation of a 
feature extraction section; 

FIG. 4 is a view showing an example of an extracted 
program feature; 

FIG. 5 is a flow chart showing the operation of a 
search engine; 

FIG. 6 is a view showing an example of a search 
result; 

FIG . 7 is a flow chart showing the operation of a link 
section; 

FIG. 8 Is a view showing an example of a display 
window generated by the link section; 
FIG . 9 is a view shovying an example of CM display; 
FIG 10 is a view showing another example of CM 

display; 

FIG. 11 is a block diagram showing the first modifi- 
cation of the first embodiment applied to a server/ 
client system; 

FIG. 12 is a block diagram showing the second 
modification of the first embodiment applied to a 
server/client system; 

FIG. 1 3 Is a block diagram showing the third modi- 
fication of the first embodiment applied to a server/ 

client system; 

FIG. 14 is a block diagram showing the basic ar- 
rangement of an information providing apparatus 
according to the second embodiment of the present 
invention; 

FIG. 1 5 is a block diagram showing the first modifi- 
cation of the second embodiment applied to a serv- 
er/client system; 

FIG. 16 is a block diagram showing the second 
modification of the second embodiment applied to 
a server/client system; 

FIG. 17 is a block diagram showing the third modi- 
fication of the second embodiment applied to a 

server/client system; 

FIG. 18 is a flow chart showing the operation of a 
viewing history recording section; 
FIGS. 1 9A and 1 9B are views showing an example 
of a viewing history; and 

FIG. 20 is a flow chart showing the user profile up- 
date operation. 

[0023] A preferred embodiment of an information pro- 
viding apparatus according to the present invention will 
now be described with reference to the accompanying 
drawings. 
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First Embodiment 

[0024] FIG. 1 is a block diagram showing the basic 
arrangement of an information providing apparatus ac- 
cording to the first embodiment of the present invention. 
This apparatus has a multimedia information database 

101 , commercial message (CM) database 108, program 
feature database 103, CM feature database 109, and 
user profile database 104. which are constructed by me- 
dia capable of random access. The databases are clas- 
sified for the descriptive convenience. Physically, one 
database may be used. 

[0025] The multimedia information database 101 
stores a number of multimedia information to be provid- 
ed. The CM database 1 08 stores a number of CM Infor- 
mation to be provided together with free programs. A 
CM feature representing the contents of CM information 
is stored in the CM feature database 109 for every CM 
information. 

[0026] The pieces of multimedia information are a 
number of programs provided by an information provid- 
er such as a broadcasting station or the Internet. Analog 
data is converted into digital data in advance and then 
stored in the multimedia information database 101 and 
managed. The digital data can be MPEG-2 compressed 
data or DV compressed data. The multimedia informa- 
tion have "title names' In units of programs and "frame 
numbers" in units of frames in each program and are 
stored in a medium, e.g., a hard disk which can be ac- 
cessed from an arbitrary position in accordance with a 
given title name and frame number. The medium is not 
limited to the hard disk and may be another medium 
such as a DVD-RAM (ROM) capable of random access. 
The multimedia information need not maintain the im- 
age size and quality of the original analog data. A com- 
pression scheme such as MPEG-1 or MPEG-4 that 
saves the image capacity may be employed depending 
on the application intended. 

[0027] The output from the multimedia information da- 
tabase 101 is supplied to a feature extraction section 

102. The feature extraction section 102 performs pre- 
determined analysis for all information held in the mul- 
timedia information database 101 , sorts the information 
in accordance with the analysis result, and adds pro- 
gram features representing the contents in units of sorts 
(e.g., in units of frames). The program features are man- 
aged by the program feature database 103 in units of 
sorts. 

[0028] CM features (CM program units) are known in 
advance. When CM information is stored in the CM da- 
tabase 108, a corresponding CM feature is stored in the 
CM feature database 109. However, a program feature 
is obtained by storing new program information in the 
multimedia information database 101. reading out the 
information from the database 101, and then analyzing 
the information. The program feature may be separately 
obtained and input to the program feature database 1 03 
by the operator using a keyboard 110. When both the 
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automatic program feature analysis and the determina- 
tion by the operator are used, a more appropriate fea- 
ture can be added to the program information (addition 
of an index). 

£ [0029] The feature extraction section 102 performs 
video analysis and acoustic/speech analysis for multi- 
media information. 

[0030] For video analysis, a technique of determining 
the video data structure on the basis of information of a 

10 cut with an instantaneous change in a video scene or 
camera movement (pan or zoom) using moving image 
analysis that has conventionally been studied, and ob- 
taining the feature of the video data can be used. 
[0031] The position where the scene instantaneously 

^5 changes can be detected by comparing the similarity be- 
tween frame images of the video data. The similarity can 
be obtained by calculating the histogram of the frequen- 
cy of a color in each image and comparing the histo- 
grams. A portion with low similarity is a point where the 

20 scene instantaneously changes. 

[0032] To provide a camera movement parameter, op- 
tical flows representing the positions of movement of 
pixels are obtained trom two images. Assuming that 
most optical flows are obtained from the background, 

25 the movement of the camera is calculated on the basis 
of dominant optical flows. 

[0033] When the camera is panning, most optical 
flows appear parallel to each other. When the camera 
is zooming, optical flows point in the direction of a certain 

30 point. Details are described in reference (1), Hirotada 
Ueno, Takafumi Miyabu, and Satoshi Yoshizawa, "Pro- 
posal of Interactive Video Editing Scheme Using Rec- 
ognition Technology". lECE Papers (D-ll), VOL. J75-D- 
II, No. 2, pp. 216 - 225 and reference (2), Masahiro Shi- 

35 bata, 'Video Contents Description Model and Its Appli- 
cation to Video Structuring", lECE Papers (D-ll), VOL. 
J78-D-II, No. 2, pp. 754 - 764. 

[0034] With acoustic/speech analysis, music and hu- 
man voice can be separated from each other because 

40 music has few mute portions and frequency compo- 
nents that are absent in human voice, and voice data 
can be discriminated because human voice has charac- 
teristic features reverse to those of music, and male 
voice and female voice have a pitch difference. 

45 [0035] Details of the method of identifying male voice 
and female voice are described in reference (3). Keiichi 
Minami, Akihito Akutsu. Hiroshi Hamada, and Yoshino- 
bu Sotomura, "Video Indexing Using Sound Information 
and Its Application", lECE Papers (D-ll), VOL. J81-D-II. 

50 No. 3, pp. 529 - 537, and a detailed description thereof 
will be omitted. 

[0036] With th is method, video data is sorted from the 
video information and speech information, and a feature 
can be added to each sort. 
55 [0037] For example, sound data is analyzed to sepa- 
rate a music portion from a portion of male/female voice. 
Then, a video scene associated with the sound data is 
discriminated into a scene associated with the music 
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portion, a scene associated v/iih male voice, and a 
scene associated with fennale voice, and features are 
deternnined for the respective scenes. 
[0038] If character data associated with video data ac- 
connpanies the video data, the text is analyzed to deter- 
mine the feature. In the U.S.A., video data contains 
character data called a closed caption. If such data can 
be used, text analysts using the conventional natural 
language processing technology can be performed to 
determine the feature according to the contents. 
[0039] That is, on the basis of character data accom- 
panying an image, a feature based on the analysis result 
of character data contents associated with video data 
can be added in units of sorts. 

[0040] The user profile database 104 is a file in which 
information (user profile) of the taste or the field of inter- 
est of each user is registered, and managed in units of 
users. The user profile is prepared by inquiring the user 
or obtaining information through a questionnaire in ad- 
vance. As shown in FIGS. 2A to 2E, the user profile has 
text information and includes keywords representing the 
taste and the field of interest of a user. FIG. 2A shows 
taste information associated with the type of programs, 
FIG. 2B shows taste information associated with the 
contents of programs, FIG. 2C shows taste information 
associated with production of programs, FIG. 2D shows 
the personal profile, and FIG. 2E shows keywords/key 
phrases representing the taste. FIG. 2A shows informa- 
tion of the program categories or genres such as sus- 
pense, drama, documentary, sports, variety^ and news. 
FIG. 28 shows the types of scenes in one program. For 
example, a movie has Information of favorite scenes 
such as an action scene, love scene, and climax scene. 
A news has information of politics, economy, sports, city 
news, and the like. FIG. 2C shows information of per- 
sons who produce programs, e.g., movie directors, ac- 
tors, actresses, composers of music used in movies, lyr- 
ic writers, and arrangers. Information of production are- 
as are also included. FIG. 2D shows personal informa- 
tion such as the age, sex, occupation, birthplace (or 
home town), and birthday. FIG. 2E shows keywords and 
key phrases representing the taste of the user, e.g., var- 
ious keywords and key phrases including a favorite 
food, favorite matter, favorite sport, hobby, and favorite 
proverb. In FIGS. 2A to 2E, a number in parentheses 
represents the number of items of taste. If a plurality of 
items are listed, the items are arranged in the order of 
priority. 

[0041] A search engine 105 searches the program 
feature database 103 and ChA feature database 109 to 

select a feature matching the user profile in the user pro- 
file database 104. This makes it possible to find out the 
information portion of user's interest. To search for that 
portion, a matching feature is detected on the basis of 
the keywords in the user profile. In this keyword match- 
ing, features matching keywords similar to the user pro- 
file can also be detected using a thesaurus (dictionary 
of synonyms or taxonomy, or index for information 
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search). The thesaurus also includes a dictionary con- 
sidering differences in usages of language between 
countries or areas or gaps between sexes or genera- 
tions (i.e., dictionary for eliminating the differences or 

5 gaps). 

[0042] With the search engine 105, associated video 
data can be finely specifically identified/searched in 
units of scenes, units associated with speech data, or 
units associated with character data, so a partial image 
10 of each user's interest can be selected and extracted. 
[0043] The search engine 105 supplies the search re- 
sult to a link section 106. 

[0044] Tho link section 1 06 processes the information 
to reproduces it. That is. the link section 106 associates 

15 the information in the CM database 108 and the result 
of search and reproduce the partial image according to 
the user profile 

[0045] A display section 107 displays the image re- 
constructed by the link section 1 06. The display section 
20 1 07 includes a loudspeaker for reproducing music infor- 
mation. 

[0046] An outline of the basic arrangement of this sys- 
tem has been described above. 

[0047] Methods of implementing the individual 
25 processing will be described below in detail. 

[0048] Details of processing by the search engine 1 05 
will be described with reference to FIG. 3. FIG. 3 shows 
the flow of processing so as to explain details of 
processing by the feature extraction section 102. 
30 [0049] The feature extraction section 1 02 can analyze 
all multimedia information stored in the multimedia in- 
formation database 10T analyze each information not 
in units of programs but in units of frames, and obtain a 
feature. 

35 [0050] Multimedia information contains not only im- 
age data but also sound and text data. Hence, analysis 
of multimedia information is performed in three steps: 
text analysis, moving image analysis, and acoustic/ 
speech analysis. The processing order is not particularly 

40 limited. 

[0051] For text analysis, closed caption information in 
the video data is extracted (steps SI and S2), mor- 
phemes are analyzed (step S3), and keywords are an- 
alyzed on the basis of the morpheme analysis result 

45 (step S4). This analysis is performed for all video pro- 
grams in the multimedia information database 101 . 
[0052] For moving image analysis, a cut of a moving 
image in video data is detected (steps S1 and S5), the 
camera movement parameter is extracted (step S6), 

50 and the video data is segmented on the basis of the 
camera movement parameter (step S7). This analysis 
is performed for all video programs in the multimedia 
information database 101 . 

[0053] For acoustic/speech analysis, acoustic identi- 
55 ficatlon is performed in video data (steps SI and SB), 
speech recognition is performed (step S9), and key- 
words are extracted on the basis of the recognition result 
(step 810). This analysis is performed for all video pro- 
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grams in the multimedia information database 101. 
[0054] Text analysis, moving Image analysis, and 
acoustic/speech analysis produce analysis results. 
[0055] By video analysis according to these proce- 
dures, various feature information are obtained in asso- 
ciation with the multimedia information. The pieces of 
feature information are processed by high-level integra- 
tion processing (step S11) of integrating the individual 
information.' 

[0056] For text analysis, moving image analysis, and 
acoustic/speech analysis, conventionally known analy- 
sis technologies can be used, as has already been de- 
scribed above. 

[0057] For example, in text analysis, a closed caption 
contained in video data is extracted, and the roles of 
words are analyzed by morpheme analysis. An impor- 
tant keyword such as a proper noun describing a scene 
is extracted from the words. As the keyword, not only a 
proper noun but also information representing a high fre- 
quency of occurrence is also used. 
[0058] In moving image analysis, video data is seg- 
mented by extracting a scene with an abrupt change or 
camera movement information (reference (1)). In 
acoustic/ speech analysis, music data and speech data 
are separated by speech identification, male voice and 
female voice are separated by speech recognition (ref- 
erence (3)), and a keyword is extracted using speech 
recognition. 

[0059] Integration processing aims at storing feature 
information obtained by the Individual processing as a 
database in association with each other and integrating 
the feature information to generate new feature informa- 
tion. 

[0060] For example, processing of associating indi- 
vidual processing is performed in the following way. 
[0061] Assume that processing is to be performed in 
units of segmented video data, and a keyword as an im- 
portant proper noun Is present in the video data. Even 
when the keyword is obtained from the caption (com- 
ment or explanation), video frames corresponding to the 
position of the keyword cannot be accurately known. 
[0062] The position of the keyword is identified using 
speech recognition, and the keyword is added to a par- 
tial image at a position with consecutive speech data as 
a feature. 

[0063] The analysis result is generated as a table as 
shown in FIG. 4. In the table shown in FIG. 4. the title 
of the program is "news", keywords as features repre- 
senting the characters or situation are "politics", "econ- 
omy", and "weather forecast", and "0:00 - 0:05', "0:15 - 
0:16", and "0:23 - 0:25" are picked up as window ap- 
pearance times (frames) associated with the respective 
keywords. That is, video data is segmented in reference 
to time (frames) in units of program titles, and important 
keywords (features) appearing In the frames are added 
to form a table. 

[0064] Details of processing by the search engine 1 05 
will be described next with reference to FIG. 5. Search 



associated with program information will be described 
below. This also applies to search associated with C\\A 
information. The search engine 105 looks up informa- 
tion in the program feature database 103 and user pro- 

5 file database 104 to extract features of user's interest, 
thereby selecting corresponding partial video data. 
[0065] Keywords are selected from the user profile 
database 104 one by one, and associated words are 
picked up using the thesaurus dictionary (steps S21 and 

10 S22). 

[0066] After picking up the associated words, the 
picked up associated words are compared with key- 
words represented In the features stored in the program 
feature database 103. If a word and keyword match with 

15 each other, information representing the position of the 
partial video data and the title to which the frame be- 
longs is recorded (steps S23, S24 and S25). In keyword 
matching, if the same associated word recurs, it is com- 
pared upon each occurrence. 

20 [0067] Processlngby the search engine 105 has been 
described above in detail. 

[0068] FIG. 6 shows an information example of partial 
video data acquired by keyword matching and regis- 
tered in step S25. In this case, one keyword in the user 

25 profile database 104 is "shopping". Information of shop- 
ping are searched for using thesaurus data, and key- 
words such as "department store", "bakery", etc., are 
selected and compared with keywords in the program 
feature database 103 to provide information of corre- 

30 spending video data as shown In FIG. 6. 

[0069] The above description has been made about 
only selection of multimedia information. CM informa- 
tion can be selected in the same way as described 
above. 

35 [0070] Details of the link section 1 06 will be described 
next with reference to the flow chart shown in FIG. 7. 
The link section 106 obtains information shown in FIG. 
6 from the search engine 1 05 and constructs, from these 
information, a display window (Index window) for pro- 

40 viding information. In this case, thumbnail images as 
shown in FIG. 8 are displayed. 

[0071] First, it is determined whether processing for 
all keywords is ended (in the example shown in FIG. 8, 
keywords are "shopping", "public facility", "transporta- 

45 tion/bank", and "health/hospital"). If processing is not 
ended, processing is continued (step S31 ). Partial video 
data selected in association with one keyword is ac- 
quired from the multimedia information database 101 
(step S32). To acquire partial video data from the multl- 

so media information database 101 by random access at 
a sufficiently high speed, time code information (frame 
information) can be directly used. Otherwise, a copy or 
partial video data, partial video data with a reduced win- 
dow size, or a copy of partial video data using a different 

55 compression ratio or compression scheme is acquired. 
[0072] One or a plurality of frames of acquired partial 
video data are acquired as representative images (step 
S33) and used as materials to construct the window. The 
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feature of each representative image is associated with 
the representative innage, and the representative image 
is associated with the partial video data (steps S34 and 
S35). Intornnalion of the representative Image is de- 
scribed using the HTML (step S36). £ 
[0073] When partial video data selected in accord- 
ance with a keyword is processed^ the next keyword is 
processed Otherwise, the above processing is repeat- 
ed (step S37). 

[0074] It is determined whether processing for all key- io 
words is ended (step S31 ). If processing for all keywords 
is ended, the contents described by the HTML are sent 
to the output or display section (step S38). Otherwise, 
processing is continued. 

[0075] FIG. 8 shows an example of a window gener- 15 
ated by the link section 106 in the above-described man- 
ner. In this example, keywords in the user profile are 
"shopping", "public facility", "transportation/bank", and 
"health/hospital". Therefore, partial video data of pro- 
grams associated with words such as "department 20 
store" and "bakery" associated with "shopping" are ac- 
quired, and representative images each forming one 
frame of partial video data are pasted in line like indices. 
CMs arranged sporadically are advertisements of spon- 
sors. CMs can also be selected on the basis of matching 25 
between program features and the user profile, like pro- 
gram information. Hence, CMs best associated with the 
selected multimedia information can be selected. In the 
window shown in FIG. 8, each representative image is 
linked to corresponding partial video data such that the 30 
partial video data is displayed by a click button. 
[0076] To generate such a window, a necessary de- 
scription is prepared using HTML. HTML is an abbrevi- 
ation for HyperText Markup Language, which indicates 
a page description language used as the general format 35 
of information provided by the WWW or W3 (World Wide 
Web) service of the Internet. HTML is based on SGML 
(Standard Generalized Markup Language) and can des- 
ignate the logical structure of a document and link be- 
tween documents by inserting a markup called a "TAG" 40 
in the document. 

[0077] WWW is a client/server information service in 
the Internet. A network user can access information us- 
ing a dedicated Web browser. Provided information are 
HTML documents called homepages, Web pages, or 45 
\N\N\N pages connected by hyper link. Information can 
be displayed by tracking the link. 
[0078] Documents handled by WWW can include 
multimedia information, and the server side can execute 
a program to perform special processing therefore. This so 
function can be used to provide a unique information 
search service. 

[0079] In the above-described example, HTML docu- 
ments are used to display CMs together with selected 
programs. A method of displaying CMs when displaying S5 
selected programs or part of programs as video data will 
be described next. 

[0080] In the example shown in FIG. 9, a CM banner 
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advertisement is displayed together with program video 
data. While displaying video data, a banner advertise- 
ment is displayed on the lower side of the program video 
data. In this case as well, the window can jump to a cor- 
responding WWW page in accordance with an instruc- 
tion from the user. 

[0081] As shown in FIG. 10, CM video data may be 
displayed as a subwindow of the program video data 
window, in this method as well, the linked WWW page 
can be used. 

[0082] As described above, since a CM best associ- 
ated with a scene of a program or a CM associated with 
user's taste can be selected in accordance with key- 
words associated with a scene of the selected program 
or user's taste, an advertisement can be effectively dis- 
played. 

[0083] As described above, according to this embod- 
iment, at least one of moving image analysis, acoustic/ 
speech analysis, and text analysis is applied to the da- 
tabase storing multimedia information and multimedia 
information provided from the database, the multimedia 
information are sorted on the basis of the analysis result, 
and the analysis result is managed in units of sorts. The 
analysis result is searched in accordance with the user 
profile, partial information of multimedia information ac- 
cording to user's taste are selected, and the selected 
partial images are associated with each other, recon- 
structed, and provided to the user. 
[0084] According to this embodiment, an information 
providing method and apparatus and an information re- 
ception apparatus capable of appropriately selecting 
and providing only portions of user's actual interest from 
a number ot multimedia information instead of filtering 
in units of programs are provided. This eliminates a dis- 
advantage of the prior art in which even a program that 
the user wants to watch only partially need be entirely 
recorded or watched. 

[0085] Since CMs can also be stored in a database 
like program information and selected like program in- 
formation, viewers surely watch CMs that the program 
provider wants the viewer to watch. In addition, since 
CMs are selected in accordance with user's taste or in- 
terest, the effect of the advertisements can be in- 
creased. This eliminates a disadvantage of the prior art 
in which when only part of video data is extracted for 
recording or watching, CMs that the program provider 
wants the viewer to watch are omitted. 
[0086] Other embodiments of the present invention 
will be described below. The same reference numerals 
as in the first embodiment denote the same parts in the 
following embodiments, and a detailed description 
thereof will be omitted. 

[0087] When the broadcasting receiving device of a 
user performs the series of processing operations of the 
first embodiment, i.e., processing of analyzing multime- 
dia information stored in a database, managing features 
of program information as the analysis result using a da- 
tabase, reading out partial information of appropriate 
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multimedia information from the database in accord- 
ance with the user profile, associating them with each 
other, and reconstructing and providing them, the 
processing amount is too large to result in overload. To 
solve this problem, a server/client system is built to per- 
form some processing operations in the server. 
[0088] FIG. 11 shows a server/client system accord- 
ing to the first modification of the first embodiment. In 
this case, the multimedia information database 1 01 , CM 
database 108, feature extraction section 102, program 
feature database 1 03, CM feature database 1 09, search 
engine 105, link section 106, and keyboard 110 are on 
the server side, and the display section 1 07, user profile 
database 104. and keyboard 111 are on the client side. 
[0089] FIG. 12 shows a server/client system accord- 
ing to the second modification of the first embodiment. 
In this case, the multimedia information database 101. 
CM database 108, feature extraction section 102, pro- 
gram feature database 103, CM feature database 109, 
search engine 1 05, and keyboard 1 1 0 are on the server 
side, and the link section 106, display section 107, user 
profile database 104, and keyboard 111 are on the client 
side. 

[0090] FIG. 13 shows a server/client system accord- 
ing to the third modification of the first embodiment. In 
this case, the multimedia information database 101, CM 
database 108, feature extraction section 102, program 
feature database 103, CM feature database 109, and 
keyboard 110 are on the server side, and the search en- 
gine 105. link section 106, display section 107, user pro- 
file database 104, and keyboard 111 are on the client 
side. 

[0091] When the system of the first embodiment is 
constructed using a server/client system, only a section 
which stores a user profile and a section which sends it 
to the server and a section which receives a search re- 
sult from the server and a section which displays it are 
on the client side, as shown in FIG. 11 . Alternatively, a 
link section being based on the basis of the search result 
is also arranged on the client side, as shown in FIG. 12, 
or a searching section is alsoarranged on the client side, 
as shown in FIG. 13. The range of functions provided 
on the client side depends on the processing capability 
of the client. When alt sectbns except the databases 
101. 103, 108, and 109 and feature extraction section 
102 are on the client side, as shown in FIG. 1 3, the client 
must download the processing result. Hence, In this ar- 
rangement, the range of functions depends on not only 
the processing capability of the client but also the infor- 
mation storage capability and line capability for down- 
load. However, since processing operations can be dis- 
tributed, this arrangement is effective when the line is a 
CATV, an optical fiber, or intranet. 

Second Embodiment. 

[0092] In the present invention, the features of video 
data can be managed not in units of programs but in 
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units of frames. This enables rating survey in units of 
frames and solves a problem of conventional rating sur- 
vey in units of programs. Hence, viewing history usable 
for analysis of user's taste or interest can be obtained. 

5 [0093] FIG. 14 is a block diagram showing the system 
arrangement of the second embodiment capable of re- 
cording viewing history in units of frames. This system 
has, in addition to the system of the first embodiment 
shown in FIG. 1, a viewing information control section 

10 120, history information recording section 121, repro- 
duction control section 122, and link section 123. The 
viewing information control section 120 acquires, from 
the reproduction control section 122. viewing informa- 
tion when a user has watched an information program 

fs (media) and transmits the information to the link section 
123. The link section 123 reads out a program feature 
corresponding to the viewing information from the pro- 
gram feature database 103. corresponds them with 
each other, and supplies them to the history information 

20 recording section 121. The viewing information contains 
Information of the watched multimedia information and 
information representing the scenes watched and the 
number of watches. 

[0094] The history information wherein the viewing in- 

25 formation and the program feature are corresponded to 
each other may be recorded in the recording section 121 
and simultaneously uploaded to the program provider 
(server) side. Alternatively, the history information may 
be uploaded to an extemal database (database of the 

30 manager on the server side) when the history informa- 
tion for a predetermined period or a predetermined 
amount of the history information is recorded in the his- 
tory information recording section 121. The viewing in- 
formation and the video data may be corresponded to 

35 each other to provide the history information represent- 
ing only watched video data, watched time sections, and 
frequencies of watches. Alternatively, using section in- 
formation of a scene of video data or index information, 
statistical information of watched scenes and frequen- 

40 cies of watches may be acquired to provide the history 
information. 

[0095] In a server/client system, the history informa- 
tion recording section 121 is preferably located in the 
same site as that of the viewing Information control sec- 

45 tion 120. However, various changes and modifications 
can be made as will be described below. 
[0096] FIG. 15 shows a server/client system accord- 
ing to the first modification of the second embodiment. 
In this case, the multimedia information database 101, 

50 CM database 108, feature extraction section 102, pro- 
gram feature database 103, CM feature database 109, 
search engine 105, link section 106, link section 123, 
and keyboard 1 1 0 are on the server side, and the display 
section 107, user profile database 104, viewing informa- 

55 tion control section 120, history information recording 
section 121, reproduction control section 122, and key- 
board 1 1 1 are on the client side. 

[0097] FIG. 16 shows a server/client system accord- 
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ing to the second modification of the second embodi- 
ment. In this case, the multimedia information database 
101, CM database 108, feature extraction section 102, 
program feature database 103, CM feature database 
109, search engine 105, and keyboard 110 are on the 
server side, and the display section 1 07, user profile da- 
tabase 104, viewing information control section 120, his- 
tory informalion recording section 121, reproduction 
control section 122, link section 106, link section 123, 
and keyboard 111 are on the client side. 
[0098] FIG. 17 shows a server/client system accord- 
ing to the third modification of the second embodiment. 
In this case, the multimedia information database 101, 
CM database 108, feature extraction section 102. pro- 
gram feature database 103, CM feature database 109, 
and keyboard 1 1 0 are on the server side, and the display 
section 107, user profile database 104, viewing informa- 
tion control section 120, history information recording 
section 121, reproduction control section 122. link sec- 
tion 106, link section 123. search engine 105, and key- 
board 111 are on the client side. 

[0099] FIG. 18 shows the processing flow of the link 

section 123. When the viewing information of a viewer 
is supplied from the reproduction control section 122, 
the viewing information control section 120 holds the 
start time and end time of watch. Even when the channel 
is changed, the viewing information control section 1 20 
determines an end and start of watch are generated. 
When watch is ended, the end is detected (step S41), 
and the watch start time and the watch end time paired 
with the start time are acquired (step S42). The informa- 
tion may be directly sent to the multimedia information 
database 101 or history information recording section 
121 and recorded (step S43). In that case, the recording 
data is recorded using the ID of the video data, start 
time, and end time, as shown in FIG. 19A. As another 
method, scenes of video data, which were viewed, are 
acquired by the link section 106 (step S44). As shown 
in FIG. 19B, a video data ID and a scene ID are proc- 
essed to provide information of frequency of watches, 
and the information is sent tothe multimedia information 
database 101 or history information recording section 
121 and recorded. The totalization method is not limited 
to this. The frequency may be calculated for each genre 
or each keyword independently of video data IDs or pro- 
gram IDs. The frequency can be calculated by counting 
"1 " when the user has watched a scene once or weight- 
ing it in accordance with the length of time of watch. 
[0100] With this processing, the audience behavior 
can be grasped in units of frames, and user's taste or 
interest can be surely known. 

[0101] Therefore, the information in the user profile 
database 1 04 may be updated on the basis of the history 
information recorded in the history information recording 
section 121. FIG. 20 shows this processing flow. First, 
the history information recording section 121 extracts a 
scene with a high (requency of watches from history in- 
formation of a user (step S50), and a feature (keyword) 



corresponding to the scene is extracted (step S51 ). It is 
determined whether the profile data of the user in the 
user profile database 1 04 has a corresponding keyword 
(item) (step S52). If YES in step S52, the pnority of the 
5 keyword in the user profile data is raised (step S53). If 
NO in step S52, the item is added to the user profile data 
(step S54). 

[0102] As described above, according to the second 
embodiment, a program and scenes thereof, which are 
watched by the user, and the number of watches are 
recorded as history information simultaneously as the 
viewer watches the program. Since the user profile is 
rewritten in accordance with the history information, a 
user profile that appropriately reflects user's taste and 
75 interest can be obtained, and information of user's inter- 
est can be selectively provided to the user. The history 
information can be acquired not in units of programs but 
in units of scenes of a program. 

Therefore, the relationship between user's taste and the 

20 scenes and contents of the program can be analyzed in 
detail. When the history information is automatically up- 
loaded from the client side to the server side, cumber- 
some acquisition can be automatically performed. 
[0103] As has been described above, according to the 

25 present invention, only video data of portions which are 
actually required by the user who is watching the pro- 
gram can be recorded or reproduced without recording 
or reproducing the entire program. In addition, partial 
video data (video data in units of sorts) are associated 

30 with each other and reconstructed to result in visually 
convenient display. Furthermore, commercial messag- 
es are also selectively provided in accordance with us- 
er's taste. Hence, even when only part of video data is 
selected and recorded or watched, commercial mes- 

35 sages that the program provider wants the viewer to 
watch are not omitted, unlike the prior art. 

Claims 

40 

1. An information providing method characterized by 
comprising: 

adding program feature data to multimedia in- 
45 formation in units of parts of the multimedia in- 

formation to form a program database; 
searching for partial information which accords 
with user profile data from said multimedia in- 
formation based on matching between the user 
50 profile data and the program feature data; and 

providing the searched partial information to a 
user. 

2. The method according to claim 1 , characterized by 
55 further comprising: 

adding commercial feature data to commercial 
information to form a commercial database; 
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55 



9 



BNSDOCID: <EP 101 6991 A2J_> 



17 

and 

providing, to the user, commercial information 
which accords with the user profile data based 
on matching between the user profile data and 
the commercial feature data when providing 
said searched partial Information to the user. 

3. An information providing apparatus characterized 
by comprising: 

a first database (101) configured to store mul- 
timedia information; 

an analyze section (102) configured to analyze 
said multimedia information stored in said first 
database using at least one analysis method of 
moving image analysis, acoustic/speech anal- 
ysis, and text analysis; 

a second database (103) configured to store 
program feature data which is obtained in units 
of parts of the multimedia information or exter- 
nally inputted: and 

a search engine (105) configured to search for 
program feature data from said second data- 
base in accordance with user profile data, and 
select partial information from said multimedia 
information stored in said first database in ac- 
cordance with searched program feature data. 

4. The apparatus according to claim 3, 
characterized by further connprising a link section 
(106) configured to obtain a representative image 
of said partial information, and construct a display 
image including said representative image and 
searched program feature data. 

5. The apparatus according to claim 3, wherein said 
user profile data includes information associated 
with the user's taste. 

6. The apparatus according to claim 3, 
characterized by further comprising a keyboard 
(110) configured to input said program feature data 
to said second database. 

7. The apparatus according to claim 3, 
characterized by further comprising a third data- 
base (104) configured to store said user profile da- 
ta. 

8. The apparatus according to claim 3, 
characterized by further comprising a fourth data- 
base (109) configured to store commercial mes- 
sage information and a fifth database configured to 
store commercial feature data, 

wherein said search engine (105) searches 
for the commercial feature data from said fifth data- 
base in accordance with the user profile data, and 
searches for the commercial message information 



IS 

corresponding to a searched commercial feature 
data from said fourth database. 

9. The apparatus according to claim 5, wherein the us- 
5 er profile data include information representing one 

of a producer, title, character, and genre of the mul- 
timedia information. 

10. The apparatus according to claim 5, wherein said 
10 search engine (105) searches for program feature 

data from said second database, and data which 
matches a thesaurus of the program feature data. 

11. The apparatus according to claim 3, 

is characterized by further comprising a history re- 

cording section (121) configured to record a viewing 
history data of a user. 

12. The apparatus according to claim 11 , wherein said 

20 viewing history data represents a user, start and 
end time of watch, and program feature data of in- 
formation watched by the user. 

13. The apparatus according to claim 11 , 
25 characterized by further comprising: 

a third database (104) configured to store said 
user profile data; and 

a rewrite section (121) configured to rewrite the 
30 user profile data stored in said third database 

in accordance with said viev;ing history data. 

14. The apparatus according to claim 3, 
characterized by further comprising a display sec- 

35 tion (107) configured to display the partial informa- 
tion selected by said search engine. 

15. The apparatus according to claim 8, 
characterized by further comprising a display sec- 
tion (107) configured to display the partial informa- 
tion selected by said search engine and display the 
commercial message information selected by said 
search engine as a banner. 

16. The apparatus according to claim 8, 
characterized by further comprising a display sec- 
tion (107) configured to display the partial informa- 
tion selected by said search engine and display the 
commercial message information selected by said 
search engine as a subwindow. 
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17. An information reception apparatus connected to 
an information providing server having a database 
which stores multimedia information and program 
55 feature data which is an analysis result of at least 
one of moving image analysis, acoustic/speech 
analysis, and text analysis or externally inputted, 
comprising; 
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a search engine (1 05) configured to search tor 
predetermined program feature data from said da- 
tabase and select partial information from said mul- 
timedia information stored in said database in ac- 
cordance with searched program feature data. s 

18. The apparatus according to claim 17, 
characterized by further comprising: 

a link section (106) configured to obtain a rep- 
resentative image of said partial information, and fO 
construct a display image including said represent- 
ative image and the searched program feature data. 

1 9. An information describing method characterized by 
comprising: ^5 

classifying information items into plural groups 
of information items relating to personal infor- 
mation of a user, some of the groups of infor- 
mation items including plural subgroups; and 20 
describing each information items in the group 
or the subgroup in an order according to a pri- 
ority of the information item which is determined 
for each user. 

25 

20. The information describing method according to 
claim 19, wherein said group of Information Items 
including data indicating personal profile of the user. 

21. The information describing method according to 30 
claim 1 9, wherein said group of information items 
including data indicating taste of the user. 
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TASTE 

INFORMATION 
ASSOCIATED WITH 
TYPE OF PROGRAM 



1) SUSPENSE 

2) DRAMA 

3) DOCUMENTARY 

4) SPORTS 

5) VARIETY 

6) NEWS 



F I G. 2A 



PERSONAL 
PROFILE 



1) SEX: MALE 

2) AGE: 38 YEARS 

3) UNMARRIED 

4) HEIGHT: 170 CM 

5) HOME TOWN: 
NARA PREFECTURE 

6) birthday: 
february 2, 1961 



TASTE 

INFORMATION 
ASSOCIATED WITH 
CONTENTS OF 
PROGRAM 

MOVIE (3) 

1) ACTION SCENE 

2) LOVE SCENE 

3) CLIMAX SCENE 
NEWS (4) 

1) POLITICS 

2) ECONOMY 

3) SPORTS 

4) CITY NEWS 



F I G. 2B 



KEYWORD 
REPRESENTING 
TASTE 

1) FAVORITE FOOD: 
APPLE 

2) FAVORITE SPORT: 
SOCCER 

3) FAVORITE PROVERB: 
TIME IS MONEY 



F I G. 2E 



TASTE 

INFORMATION 
ASSOCIATED WITH 
PRODUCTION OF 
PROGRAM 

FILM DIRECTOR (2) 

1) ICHIRO TOSHIBA 

2) J IRQ TOSHIBA 
ACTOR (2) 

1) SABURO TOSHIBA 

2) SHIRO TOSHIBA 
ACTRESS (1) 
HANAKO TOSHIBA 
COMPOSER (1) 
GORO TOSHIBA 
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